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Why GAO Did This Study 

The Department of Homeland 
Security's (DHS) Domestic Nuclear 
Detection Office (DNDO) is testing 
new advanced spectroscopic portal 
(ASP) radiation detection monitors. 
DNDO expects ASPs to reduce 
both the risk of missed threats and 
the rate of innocent alarms, which 
DNDO considers to be key 
limitations of radiation detection 
equipment currently used by 
Customs and Border Protection 
(CBP) at U.S. ports of entry. 
Congress has required that the 
Secretary of DHS certify that ASPs 
provide a significant increase in 
operational effectiveness before 
obligating funds for full-scale 
procurement. GAO was asked to 
review (1) the degree to which 
DHS's criteria for a significant 
increase in operational 
effectiveness address the 
limitations of existing radiation 
detection equipment, (2) the rigor 
of ASP testing and preliminary test 
results, and (3) the ASP test 
schedule. GAO reviewed the DHS 
criteria, analyzed test plans, and 
interviewed DHS officials. 



What GAO Recommends 



GAO recommends that DHS assess 
ASPs against the full potential of 
current equipment and revise the 
program schedule to allow time to 
conduct computer simulations of 
ASPs' capabilities and to uncover 
and resolve problems with ASPs 
before full-scale deployment. DHS 
agreed to a phased deployment that 
should allow time to uncover ASP 
problems but disagreed with GAO's 
other recommendations. GAO 
believes its recommendations 
remain valid. 

View GAO-09-655 or key components. 

For more information, contact Gene Aloise at 

(202)512-3841 oraloisee@gao.gov. 



What GAO Found 

The DHS criteria for a significant increase in operational effectiveness require 
a minimal improvement in the detection of threats and a large reduction in 
innocent alarms. Specifically, the criteria require a marginal improvement in 
the detection of certain weapons-usable nuclear materials, considered to be a 
key limitation of current-generation portal monitors. The criteria require 
improved performance over the current detection threshold, which for certain 
nuclear materials is based on the equipment's limited sensitivity to anything 
more than lightly shielded materials, but do not specify a level of shielding 
that smugglers could realistically use. In addition, DNDO has not completed 
efforts to improve current-generation portal monitors' performance. As a 
result, the criteria do not take the current equipment's full potential into 
account. With regard to innocent alarms, the other key limitation of current 
equipment, meeting the criteria could result in hundreds fewer innocent 
alarms per day, thereby reducing CBP's workload and delays to commerce. 

DHS increased the rigor of ASP testing in comparison with previous tests. For 
example, DNDO mitigated the potential for bias in performance testing (a 
concern GAO raised about prior testing) by stipulating that there would be no 
ASP contractor involvement in test execution. Such improvements added 
credibility to the test results. However, the testing still had limitations, such as 
a limited set of scenarios used in performance testing to conceal test objects 
from detection. Moreover, the preliminary results are mixed. The results show 
that the new portal monitors have a limited ability to detect certain nuclear 
materials at anything more than light shielding levels: ASPs performed better 
than current-generation portal monitors in detection of such materials 
concealed by light shielding approximating the threat guidance for setting 
detection thresholds, but differences in sensitivity were less notable when 
shielding was slightly below or above that level. Testing also uncovered 
multiple problems in ASPs meeting the requirements for successful 
integration into operations at ports of entry. CBP officials anticipate that, if 
ASPs are certified, new problems will appear during the first few years of 
deployment in the field. 

While DNDO's schedule underestimated the time needed for ASP testing, test 
delays have allowed more time for review and analysis of results. DNDO's 
original schedule anticipated completion in September 2008. Problems 
uncovered during testing of ASPs' readiness to be integrated into operations 
at U.S. ports of entry caused the greatest delays to this schedule. DHS's most 
recent schedule anticipated a decision on ASP certification as early as May 
2009, but DHS recently suspended field validation due to ASP performance 
problems and has not updated its schedule for testing and certification. In any 
case, DNDO does not plan to complete computer simulations that could 
provide additional insight into ASP capabilities and limitations prior to 
certification even though delays have allowed more time to conduct the 
simulations. DNDO officials believe the other tests are sufficient for ASPs to 
demonstrate a significant increase in operational effectiveness. 
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Abbreviations 



ASP advanced spectroscopic portal 

CBP Customs and Border Protection 

DHS Department of Homeland Security 

DNDO Domestic Nuclear Detection Office 

DOE Department of Energy 

HEU highly enriched uranium 

PVT polyvinyl toluene 
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Congressional Requesters 

Preventing radioactive material from being smuggled into the United 
States is a key national security objective. In particular, terrorists could 
use special nuclear material such as highly enriched uranium (HEU) or 
plutonium in a nuclear weapon; other radioactive materials could be used 
in a radiological dispersal device (a "dirty bomb")- The national security 
mission of U.S. Customs and Border Protection (CBP), an agency within 
the Department of Homeland Security (DHS), includes screening for 
smuggled nuclear or radiological material while facilitating the flow of 
legitimate trade and travel. To screen cargo at ports of entry, CBP 
conducts primary inspections with radiation detection equipment called 
portal monitors — large stationary detectors through which cargo 
containers and vehicles pass as they enter the United States. When 
radiation is detected, CBP conducts secondary inspections using a second 
portal monitor to confirm the original alarm and a handheld radioactive 
isotope identification device to identify the radiation's source and 
determine whether it constitutes a threat. CBP officers must investigate 
each alarm until they are convinced that the vehicle, occupants, and any 
cargo pose no threat and can be allowed to enter the United States. 

According to DHS's Domestic Nuclear Detection Office (DNDO), the 
current generation of radiation detection equipment has limitations. 1 
Specifically, the polyvinyl toluene (PVT) portal monitors currently in use 
can detect radiation but cannot identify the source. As a result, the 
monitors' radiation alarms can be set off even by benign, naturally 
occurring radioactive material. One way to reduce the rate of such 
innocent alarms — and thereby minimize unnecessary secondary 
inspections and enhance the flow of commerce — is to adjust the 
operational thresholds for the level of radiation required for PVTs to alarm 
(i.e., operate the PVTs at a reduced level of sensitivity). However, reducing 



x DNDO was established within DHS in 2005; its mission includes developing, testing, 
acquiring, and supporting the deployment of radiation detection equipment at U.S. ports of 
entry. CBP began deploying portal monitors in 2002, prior to DNDO's creation, under the 
radiation portal monitor project. For additional information on DNDO's overall efforts to 
combat nuclear smuggling, see GAO, Nuclear Detection: Domestic Nuclear Detection 
Office Should Improve Planning to Better Address Gaps and Vulnerabilities, GAO-09-257 
(Washington, D.C.: Jan. 29, 2009). 
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the sensitivity may make it more difficult to detect certain nuclear 
materials. 

Since 2005, DNDO has been developing and testing advanced 
spectroscopic portals (ASP), a new type of portal monitor designed to 
both detect radiation and identify the source. The new portal monitors use 
technology similar to that in handheld identification devices currently 
used for secondary screening. Key differences from handheld 
identification devices include a larger number of detectors, more 
sophisticated software, and a more extensive library of radiation 
signatures that may provide more consistent and rapid screening and may 
increase the likelihood of correct identification. DNDO hopes to use the 
new portal monitors to replace at least some PVTs currently used for 
primary screening, as well as PVTs and handheld identification devices 
currently used for secondary screening. However, the new portal monitors 
cost significantly more than PVTs. We estimated in September 2008 that 
the lifecycle cost of each standard cargo version of the ASP (including 
deployment costs) is about $822,000, compared with about $308,000 for 
the PVT standard cargo portal, and that the total program cost for DNDO's 
latest plan for deploying radiation portal monitors — which relies on a 
combination of ASPs and PVTs and does not deploy radiation portal 
monitors at all border crossings — would be about $2 billion. 2 Moreover, 
CBP officials expect operation and maintenance costs to be significantly 
higher for ASPs than for PVTs because of the greater complexity of ASP 
equipment. 

Concerned about the performance and cost of the new ASP monitors, 
Congress required the Secretary of Homeland Security to certify that the 
monitors will provide a "significant increase in operational effectiveness" 
before DNDO obligates funds for full-scale ASP procurement. 3 The 
Secretary must submit separate certifications for primary and secondary 
inspection. In response, DNDO, CBP, and the DHS management 
directorate jointly issued criteria in July 2008 for determining whether the 
new technology provides a significant increase in operational 



2 GAO, Combating Nuclear Smuggling: DHS's Program to Procure and Deploy Advanced 
Radiation Detection Portal Monitors Is Likely to Exceed the Department's Previous Cost 
Estimates, GAO-08-1108R (Washington, D.C.: Sept. 22, 2008). 

Consolidated Appropriations Act, 2007, Pub. L. No. 110-161, 121 Stat. 1844, 2069 (2007); 
Consolidated Security, Disaster Assistance, and Continuing Appropriations Act, 2009, Pub. 
L. No. 110-329, 121 Stat. 3574, 3679 (2008). 
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effectiveness — four criteria for primary screening and two for secondary 
screening (see fig. 1). The primary screening criteria require that the new 
portal monitors detect potential threats as well as or better than PVTs, 
show improved performance in detection of HEU, and reduce innocent 
alarms. To meet the secondary screening criteria, the new portal monitors 
must reduce the probability of misidentifying special nuclear material 
(e.g., HEU and plutonium) and the average time to conduct secondary 
screenings. 



Figure 1 : DHS Criteria for Demonstrating a Significant Increase in Operational 
Effectiveness 



Primary screening criteria 



When special nuclear material is present 
in cargo without naturally occurring 
radioactive material, the ASP probability of 
a correct operational outcome must be 
equal to or greater than that of the PVT. 
(For HEU, ASPs must show improved 
performance compared to PVTs at 
operational thresholds.) 

When special nuclear material is present 
in cargo with naturally occurring radioac- 
tive material, the ASP in primary must 
increase the probability of a correct 
operational outcome compared to the 
current end-to-end system. 

When licensable medical or industrial 
isotopes are present in cargo, the ASP 
probability of a correct operational 
outcome must be equal to or greater than 
that of the PVT. 



When the only radioactive source present 
in the cargo is naturally occurring 
radioactive material, the ASP must refer at 
least 80 percent fewer conveyances for 
further inspection than the PVT. 



Source: DHS. 



Secondary screening criteria 



When compared to the handheld 
radioactive isotope identification device, 
ASP must reduce, by at least a factor of 
two, the probability that special nuclear 
material is misidentified as naturally 
occurring radioactive material, a 
medical/industrial radionuclide, unknown, 
or no source at all. 

When compared to the handheld 
radioactive isotope identification device, 
the ASP must reduce the average time 
required to correctly release convey- 
ances from secondary screening. 



To demonstrate a significant increase in operational effectiveness for 
either primary or secondary screening, ASPs must satisfy all of the criteria 
for that deployment option, independent of satisfying the criteria for the 
other option. The criteria generally compare the new portal monitors to 
current-generation equipment as used under CBP's standard operating 
procedure. For example, the standard operating procedure for secondary 
screening calls for inconclusive readings to be sent for additional analysis 
to CBP's Laboratories and Scientific Services, which has access to 
additional software and trained experts. 
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DNDO designed and coordinated a series of tests, originally scheduled to 
run from April 2008 through September 2008, to determine whether the 
new portal monitors meet the certification criteria for primary and 
secondary screening and are ready for deployment. Key phases of testing 
completed to date include verifying that ASPs meet DNDO's performance 
specification, which was followed by concurrent testing of the new and 
current equipments' ability to detect and identify threats and of ASPs' 
readiness to be integrated into operations for both primary and secondary 
screening at ports of entry. Two remaining phases not yet completed 
include field validation at four northern and southern border crossings and 
two seaports, as well as an independent evaluation, conducted by the DHS 
Science and Technology Directorate at one of the seaports, of the new 
portal monitors' effectiveness and suitability (see fig. 2). Two ASP vendors 
have contracts with DNDO to develop the new portal monitors and are 
participating in the round of testing that began in 2008. 4 DNDO designed 
the testing to allow each vendor's system to complete all test phases and 
be certified based on its own performance as providing a significant 
increase in operational effectiveness. 



Figure 2: Test Sequence Leading Up to ASP Certification 



Lead 
agencies 

Phases 



System 
qualification test 



Verify technical 
achievement of 
ASP performance 
specification 



DNDO 



Concurrent performance and integration testing 



Performance testing: 

Evaluate ASP performance 
for detection and 
identification of threats 




Integration testing: 

Demonstrate that ASPs 
are ready to be integrated 
into the interdiction 
systems at U.S. ports of 
entry and to advance to 
field validation 



CBP 



Field validation 



Perform system 
installation procedures 
and process, train 
CBP officers in ASP 
use, and use ASPs 
to screen cargo 



DHS Science 
and Technology 
Directorate 

Operational test 
and evaluation 



Conduct an indepen- 
dent evaluation of ASP 
effectiveness and 
suitability when 
operated by CBP 



Source: GAO analysis of DNDO information. 



4 DNDO had a contract with a third ASP vendor whose system uses a more expensive type 
of detector that must be cooled by liquid nitrogen. DNDO determined it was not in the best 
interests of the government to exercise the option on the contract and allowed it to expire 
in November 2008. The vendor's ASP did not participate in the 2008 round of testing. 
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We have raised concerns since 2006 regarding DNDO's previous efforts to 
develop and test the new portal monitors. In October 2006, we found that 
DNDO's analysis of the benefits and costs of deploying the new portal 
monitors relied on assumptions of their anticipated performance level 
instead of actual test data. 5 Among other things, we recommended that 
DNDO conduct further testing before spending additional funds to 
purchase the new equipment. In September 2007, we testified that DNDO's 
testing at the Department of Energy's (DOE) Nevada Test Site did not 
represent an objective or rigorous assessment because DNDO used biased 
test methods that enhanced the apparent performance of the ASPs and did 
not test the limitations of their detection capabilities. 6 Most recently, we 
found in September 2008 that a DNDO report on testing conducted in 2007 
did not accurately depict test results and could potentially be misleading. 7 
We concluded that the results could identify areas for improvement but 
should not be used as indicators of ASPs' overall performance. 

In this context, you asked us to review the 2008 round of testing leading up 
to the Secretary of Homeland Security's decision on ASP certification. We 
reviewed (1) the degree to which DHS's criteria for a significant increase 
in operational effectiveness address the limitations of the current 
generation of radiation detection equipment, (2) the rigor of the testing as 
a basis for determining ASPs' operational effectiveness and preliminary 
results of testing completed to date, and (3) the extent to which the test 
schedule allows time for DHS to review and analyze results. This report 
updates our September 2008 testimony, which included preliminary 
observations on the DHS criteria for a significant increase in operational 
effectiveness and the 2008 round of testing. 8 



°GAO, Combating Nuclear Smuggling: DHS's Cost-Benefit Analysis to Support the 
Purchase of New Radiation Detection Portal Monitors Was Not Based on Available 
Performance Data and Did Not Fully Evaluate All the Monitors' Costs and Benefits, 
GAO-07-133R (Washington, D.C.: Oct. 17, 2006). 

6 GAO, Combating Nuclear Smuggling: Additional Actions Needed to Ensure Adequate 
Testing of Next Generation Radiation Detection Equipment, GAO-07-1247T (Washington, 
D.C.: Sept. 18, 2007). 

7 GAO, Combating Nuclear Smuggling: DHS's Phase 3 Test Report on Advanced Portal 
Monitors Does Not Fully Disclose the Limitations of the Test Results, GAO-08-979 
(Washington, D.C.: Sept. 30, 2008). 

8 GAO, Combating Nuclear Smuggling: DHS Needs to Consider the Full Costs and 
Complete All Tests Prior to Making a Decision on Whether to Purchase Advanced Portal 
Monitors, GAO-08-1178T (Washington, D.C.: Sept. 25, 2008). 
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To conduct our review, we analyzed DHS's criteria for a significant 
increase in operational effectiveness and DNDO's written response to our 
detailed questions regarding the criteria. Because the criteria compare the 
new portal monitors to existing equipment, we analyzed the threat 
guidance used to set detection thresholds for PVTs and interviewed DOE 
and national laboratory officials responsible for the guidance. In addition, 
we analyzed the test plans for the 2008 round of testing, including the test 
schedule and reasons for any delays. We interviewed DNDO, CBP, and 
other DHS officials responsible for conducting and monitoring tests, and 
we observed 1 day each of performance testing at the Nevada Test Site 
and integration testing at DOE's Pacific Northwest National Laboratory. 
We analyzed preliminary or final results for the phases of testing 
completed during our review, and we interviewed DNDO and CBP officials 
regarding the results. (App. I presents a detailed discussion of the scope 
and methodology of our review.) 

We conducted this performance audit from May 2008 to May 2009 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. 9 



9 This report does not include certain details about the capabilities and limitations of PVTs 
and ASPs that DHS considers to be "for official use only." We have prepared a "for official 
use only" version of this report in which we include such details (GAO-09-354SU). 



Page 6 



GAO-09-655 Combating Nuclear Smuggling 



DHS's Criteria for 
Significant Increase in 
Operational 
Effectiveness Require 
a Marginal 
Improvement in the 
Detection of Certain 
Nuclear Materials and 
a Large Reduction in 
Innocent Alarms 



Although the DHS criteria for primary screening require an improved 
ability to detect certain nuclear materials at operational thresholds, ASPs 
could meet the criteria for improvement while still failing to detect 
anything more than lightly shielded material. DNDO officials acknowledge 
that passive radiation detection equipment, which includes both the new 
and current-generation portal monitors, is capable of detecting certain 
nuclear materials only when this material is unshielded or lightly 
shielded. 10 For this reason, the DOE threat guidance used to set PVTs' 
detection threshold is based on the equipment's limited sensitivity to 
anything more than lightly shielded nuclear material rather than on the 
assumption that smugglers would take effective shielding measures. DOE 
developed the guidance in 2002 and 2003 when CBP began deploying PVTs 
for primary screening. DOE and national laboratory officials responsible 
for the guidance told us the assumption of light shielding was based not on 
an analysis of the capabilities of potential smugglers to take effective 
shielding measures but rather on the limited sensitivity of PVTs to detect 
anything more than certain lightly shielded nuclear materials. In contrast, 
PVTs are more sensitive to the relatively strong radiation signature of 
other nuclear materials, and the threat guidance assumes a higher level of 
shielding for setting the operational threshold for detection of such 
materials. However, even for such materials, the DOE threat guidance 
assumes that shielding would not exceed a level provided by the contents 
of an average cargo container. 



Moreover, DNDO has not completed efforts to fine-tune PVTs' software 
and thereby improve sensitivity to nuclear materials. As a result, the 
criteria compare ASPs to the current performance of PVTs and do not take 
potential improvements into account, which affects any assessment of 
"significant" improvement over current technology. DNDO officials expect 
they can achieve small improvements to PVTs' performance through 
additional development of "energy windowing," a technique currently 
being used in PVTs to provide greater sensitivity than otherwise possible. 
Pacific Northwest National Laboratory officials responsible for developing 
the technique also told us small improvements may be possible, and CBP 
officials have repeatedly urged DNDO to investigate the potential of the 
technique. DNDO collected the data needed to further develop energy 



According to DNDO and CBP officials, active imaging techniques (e.g., radiography 
systems to provide images of the contents of cargo containers) and other measures 
complement radiation detection equipment. In particular, such measures provide the 
capability to spot smuggled nuclear materials that are too heavily shielded to be detected 
by PVTs or ASPs. 
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windowing during the 2008 performance testing at the Nevada Test Site 
but has not yet funded Pacific Northwest National Laboratory efforts to 
analyze the data and further develop the technique. 

Other aspects of the criteria for a significant increase in operational 
effectiveness require that ASPs either provide more than a marginal 
improvement in addressing other limitations of current-generation 
equipment or at least maintain the same level of performance in areas in 
which the current-generation equipment is considered adequate: 

• The primary screening requirement for an 80 percent reduction in the rate 
of innocent alarms could result in hundreds of fewer secondary screenings 
per day, thereby reducing CBP's workload and delays to commerce. The 
actual reduction in the volume of innocent alarms would vary and would 
be greatest at the nation's busiest ports of entry, such as Los Angeles/Long 
Beach, where CBP officials report that PVTs generate up to about 600 
innocent alarms per day. 11 A DNDO official said the requirement for an 80 
percent reduction in innocent alarms was developed in conjunction with 
CBP and was based on a level that would provide meaningful workload 
relief. 

• The primary screening criteria requiring that ASPs provide at least the 
same level of sensitivity to plutonium and medical and industrial isotopes, 
but not specifying an improvement, were based on DNDO's assessment 
that PVTs adequately detect such materials, which have a stronger 
radiation signature than HEU. 12 In addition, CBP officials said that 
including medical and industrial isotopes in the criteria addressed a CBP 
requirement for verifying that those transporting certain quantities of 
these materials into the United States are properly licensed. 13 

• The secondary screening requirement that ASPs reduce the probability of 
misidentifying special nuclear material by one-half addresses the inability 
of relatively small handheld devices to consistently locate and identify 



About 45 percent of all sea containers arriving in the United States come through Los 
Angeles/Long Beach. In fiscal year 2006, CBP cleared more than 5 million containers 
through the port. 

12 The criteria require an improvement when the radiation emitted by naturally occurring 
radioactive material is used to mask smuggled special nuclear material, including both 
HEU and plutonium. 

13 For additional information regarding the requirement to verify the legitimacy of 
radioactive material shipments, see GAO, Nuclear Security: NRC and DHS Need to Take 
Additional Steps to Better Track and Detect Radioactive Materials, GAO-08-598 
(Washington, D.C.: June 19, 2008). 
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potential threats in large cargo containers. For example, a handheld device 
may fail to correctly identify special nuclear material if the material is 
well-shielded or the device is not placed close enough to a radiation 
source to obtain a recognizable measurement. According to CBP and 
DNDO, the requirement for a reduction in the average time to conduct 
secondary screenings is not more specific because the time varies 
significantly among ports of entry and types of cargo being screened. 



j~)jjg IncrGclSGd. the Improvements to the 2008 round of testing addressed concerns we raised 

about earlier rounds of ASP testing. However, the testing still had 
Rigor of Advanced limitations, and the preliminary results are mixed. 

Portal Monitor Testing 



Improvements to Testing As we testified in September 2008, DHS's improvements to the 2008 round 
Provided Credibility to of ASP testing addressed concerns we raised about previous tests. A 

Test Results particular area of improvement was in the performance testing at the 

Nevada Test Site, where DNDO compared the capability of ASP and 
current-generation equipment to detect and identify nuclear and 
radiological materials, including those that could be used in a nuclear 
weapon. The improvements addressed concerns we previously raised 
about the potential for bias and provided credibility to the results within 
the limited range of scenarios tested by DNDO. For example, we reported 
in 2007 that DNDO had allowed ASP contractors to adjust their systems 
after preliminary runs using the same radiological materials that would be 
used in the formal tests. In contrast, the plan for the 2008 performance test 
stipulated that there would be no system contractor involvement in test 
execution, and no ASP contractors were at the test location on the day we 
observed performance testing. Furthermore, DNDO officials told us, and 
we observed, that they did not conduct preliminary runs with threat 
objects used in the formal tests. In 2007, we reported that DNDO did not 
objectively test the handheld identification devices because it did not 
adhere to CBP's standard operating procedure for using the devices to 
conduct a secondary inspection, which is fundamental to the equipment's 
performance in the field. DNDO addressed this limitation in the 2008 
round of performance testing: CBP officers operated the devices and 
adhered as closely to the standard operating procedure as test conditions 
allowed. While the test conditions did not allow CBP officers to obtain 
real-time technical support in interpreting the device's measurements, as 
they would in the field to increase the probability of correctly identifying a 
radiation source, DNDO officials said they addressed this limitation. For 
example, they treated a decision by a CBP officer to indicate the need for 
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technical support as a correct outcome if the test scenario involved the 
use of a potential threat, such as HEU. 

Other aspects of testing, while not specifically addressing concerns we 
previously raised, also added credibility to the test results. Based on our 
analysis of the performance test plan, we concluded that the test design 
was sufficient to identify statistically significant differences between the 
new technology and current-generation systems when there were 
relatively large differences in performance. Specifically, DNDO conducted 
a sufficient number of runs of each scenario used in the 2008 performance 
testing to identify such differences. 

With regard to the general conduct of the 2008 round of testing, two 
aspects, in particular, enhanced the overall rigor of the tests: (1) criteria 
for ensuring that ASPs met the requirements for each phase before 
advancing to the next, and (2) the participation of CBP and the DHS 
Science and Technology Directorate. 14 The test and evaluation master plan 
established criteria requiring that the ASPs have no critical or severe 
issues rendering them completely unusable or impairing their function 
before starting or completing any test phase. In addition, the criteria 
established a cumulative limit of 10 issues requiring a work-around (e.g., a 
straightforward corrective step, such as a minor change in standard 
operating procedures) and 15 cosmetic issues not affecting proper 
functioning. DNDO and CBP adhered to the criteria even though doing so 
resulted in integration testing conducted at the Pacific Northwest National 
Laboratory taking longer than anticipated and delaying the start of field 
validation. For example, DNDO and CBP did not allow a vendor's ASP 
system to complete integration testing until all critical or severe issues had 
been resolved. 

The involvement of CBP and the DHS Science and Technology Directorate 
provided an independent check, within DHS, of DNDO's efforts to develop 
and test the new portal monitors. For example, the lead CBP official 
involved in ASP testing told us that DNDO provided an initial assessment 
of the severity of issues uncovered during testing, but CBP made the final 
decision on categorizing them as critical, severe, work-around, or 
cosmetic issues. CBP also added a final requirement to integration testing 



In the case of ASP testing, the Science and Technology Directorate serves as the 
independent operational test authority, which reports directly to the DHS Under Secretary 
for Management. 
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before proceeding to field validation to demonstrate ASPs' ability to 
operate for 40 hours without additional problems. According to CBP 
officials, their efforts to resolve issues prior to field validation reflect the 
importance CBP places on ensuring that ASPs are sufficiently stable and 
technically mature to operate effectively in a working port of entry and 
thereby provide for a productive field validation. 

The DHS Science and Technology Directorate, which is responsible for 
developing and implementing the department's test and evaluation policies 
and standards, will have the lead role in the final phase of ASP testing; the 
final phase, consisting of 21 days of continuous operation, is scheduled to 
begin at one seaport after the completion of field validation. The Science 
and Technology Directorate identified two critical questions to be 
addressed through operational testing: (1) Will the ASP system improve 
operational effectiveness (i.e., detection and identification of threats) 
relative to the current-generation system, and (2) is the ASP system 
suitable for use in the operational environment at land and sea ports of 
entry? The suitability of ASPs includes factors such as reliability, 
maintainability, and supportability. Because the operational testing 
conducted at one seaport is not sufficient to fully answer these 
questions — for example, because the testing will not allow threat objects 
to be inserted into cargo containers — the directorate plans to also conduct 
an independent analysis of the results from previous test phases, including 
performance testing. 

The 2008 testing still had limitations, which do not detract from the test 
results' credibility but do require that results be appropriately qualified. 
Limitations included the following: 

• The number of handheld identification device measurements collected 
during performance testing was sufficient to distinguish only particularly 
large differences from ASPs' identification ability. In particular, the 
standard operating procedure for conducting secondary inspections using 
ASPs, which requires less time than when using handheld devices, allowed 
DNDO to collect more than twice as many ASP measurements and to test 
ASPs' identification ability against more radiation sources than used to 
test handheld identification devices. 

• The performance test results cannot be generalized beyond the limited set 
of scenarios tested. For example, DNDO used a variety of masking and 
shielding scenarios designed to include cases where both systems had 100 
percent detection, cases where both had zero percent detection, and 
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several configurations in between so as to estimate the point where 
detection capability ceased. 15 However, the scenarios did not represent the 
full range of possibilities for concealing smuggled nuclear or radiological 
material. For example, DNDO only tested shielding and masking scenarios 
separately, to differentiate between the impacts of shielding and masking 
on the probabilities of detection and identification. As a result, the 
performance test results cannot show how well each system would detect 
and identify nuclear or radiological material that is both shielded and 
masked, which might be expected in an actual smuggling incident. 
Similarly, DNDO used a limited number of threat objects to test ASPs' 
detection and identification performance, such as weapons-grade 
plutonium but not reactor-grade plutonium, which has a different isotopic 
composition. A report on special testing of ASPs conducted by Sandia 
National Laboratories in 2007 recommended that future tests use 
plutonium sources having alternative isotopic compositions. Sandia based 
its recommendations on results showing that the performance of ASP 
systems varied depending on the isotopic composition of plutonium. 

• The Science and Technology Directorate's operational testing is designed 
to demonstrate that the average time between equipment failures (the 
measure of ASPs' reliability) is not less than 1,000 hours. Thus, the testing 
will not show how reliable the equipment will be over a longer term. DHS 
Science and Technology Directorate officials recognize this limitation and 
said they designed operational testing only to demonstrate compliance 
with the ASP performance specification. Furthermore, to the extent that 
the Science and Technology Directorate relies on performance test results 
to evaluate ASPs' ability to detect and identify threats, its analysis of ASPs' 
effectiveness will be subject to the same limitations as the original testing 
and analysis conducted by DNDO. 



Preliminary Test Results The preliminary results presented to us by DNDO are mixed, particularly 
Are Mixed m * ne capability of ASPs used for primary screening to detect certain 

shielded nuclear materials. However, we did not obtain DNDO's final 
report on performance testing conducted at the Nevada Test Site until 
early April 2009, and thus we had limited opportunity to evaluate the 
report. In addition, we are not commenting on the degree to which the 
final report provides a fair representation of ASPs' performance. 
Preliminary results from performance testing show that the new portal 



°Masking is the use of naturally occurring radioactive material to make the radiation 
emitted by smuggled material appear to be caused by innocent cargo. In contrast, shielding 
blocks radiation from being emitted. 
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monitors detected certain nuclear materials better than PVTs when 
shielding approximated DOE threat guidance, which is based on light 
shielding. In contrast, differences in system performance were less notable 
when shielding was slightly increased or decreased: Both the PVTs and 
ASPs were frequently able to detect certain nuclear materials when 
shielding was below threat guidance, and both systems had difficulty 
detecting such materials when shielding was somewhat greater than threat 
guidance. DNDO did not test ASPs or PVTs against moderate or greater 
shielding because such scenarios are beyond both systems' ability. (See 
fig. 3 for a summary of performance test results for detection of certain 
nuclear materials.) 



Figure 3: Preliminary Results from 2008 Performance Testing for Detection of 
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Source: GAO analysis of DNDO information. 

Note: The specific amount and type of shielding assumed in DOE threat guidance is classified. 

With regard to secondary screening, ASPs performed better than handheld 
devices in identification of threats when masked by naturally occurring 
radioactive material. However, differences in the ability to identify certain 
shielded nuclear materials depended on the level of shielding, with 
increasing levels appearing to reduce any ASP advantages over the 
handheld identification devices — another indication of the fundamental 
limitation of passive radiation detection. 

Other phases of testing, particularly integration testing, uncovered multiple 
problems meeting requirements for successfully integrating the new 
technology into operations at ports of entry. Of the two ASP vendors 
participating in the 2008 round of testing, one has fallen several months 
behind in testing due to the severity of the problems it encountered during 
integration testing; the problems were so severe that it may have to redo 
previous test phases to be considered for certification. The other vendor's 
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system completed integration testing, but CBP suspended field validation of 
the system after 2 weeks because of serious performance problems that may 
require software revisions. In particular, CBP found that the performance 
problems resulted in an overall increase in the number of referrals for 
secondary screening compared to the existing equipment. According to CBP, 
this problem will require significant corrective actions before testing can 
resume; such corrective actions could in turn change the ability of the ASP 
system to detect threats. The problem identified during field validation was in 
addition to ones identified during integration testing, which required multiple 
work-arounds and cosmetic changes before proceeding to the next test phase. 
For example, one problem requiring a work-around related to the amount of 
time it takes for the ASP to sound an alarm when a potential threat material 
has been detected. Specifications require that ASPs alarm within two seconds 
of a vehicle exiting the ASP. However, during testing, the vendor's ASP took 
longer to alarm when a particular isotope was detected. The work-around to 
be implemented during field validation requires that all vehicles be detained 
until cleared by the ASP; the effect on commerce must ultimately be 
ascertained during field validation. 

CBP officials anticipate that they will continue to uncover problems 
during the first few years of use if the new technology is deployed in the 
field. The officials do not necessarily regard such problems to be a sign 
that testing was not rigorous but rather a result of the complexity and 
newness of the technology and equipment. 



Delays to the schedule for the 2008 round of testing have allowed more 
time for analysis and review of results, particularly from performance 
testing conducted at the Nevada Test Site. The original schedule, which 
underestimated the time needed for testing, anticipated completion of 
testing in mid-September 2008 and the DHS Secretary's decision on ASP 
certification between September and November 2008. DHS officials 
acknowledged that scheduling a certification decision shortly after 
completion of testing would leave limited time to complete final test 
reports and said the DHS Secretary could rely instead on preliminary 
reports if the results were favorable to ASPs. DHS's most recent schedule 
anticipated a decision on ASP certification as early as May 2009, but DHS 
has not updated its schedule for testing and certification since suspending 
field validation in February 2009 due to ASP performance problems. 

Problems uncovered during testing of ASPs' readiness to be integrated 
into operations at U.S. ports of entry have caused the greatest delays to 
date and have allowed more time for DNDO to analyze and review the 



Schedule Delays Have 
Allowed More Time 
for Analysis and 
Review of Test 
Results, but DNDO's 
Latest Schedule Does 
Not Include Computer 
Simulations to 
Provide Additional 
Insight into ASP 
Capabilities 
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results of performance testing. Integration testing was originally scheduled 
to conclude in late July 2008 for both ASP vendors. The one ASP system 
that successfully passed integration testing did not complete the test until 
late November 2008 — approximately 4 months behind schedule. (The 
delays to integration testing were due in large part to the adherence of 
DNDO and CBP to the criteria discussed earlier for ensuring that ASPs met 
the requirements for each test phase.) In contrast, delays to performance 
testing, which was scheduled to run concurrently with integration testing, 
were relatively minor. Both ASP systems completed performance testing 
in August 2008, about a month later than DNDO originally planned. 

The schedule delays have allowed more time to conduct injection 
studies — computer simulations for testing the response of ASPs and PVTs 
to the radiation signatures of threat objects randomly "injected" 
(combined) into portal monitor records of actual cargo containers 
transported into the United States, including some containers with 
innocent sources of radiation. However, DNDO does not plan to complete 
the studies prior to the Secretary of Homeland Security's decision on 
certification even though DNDO and other officials have indicated that the 
studies could provide additional insight into the capabilities and 
limitations of advanced portal monitors. According to DNDO officials, 
injection studies address the inability of performance testing conducted at 
the Nevada Test Site to replicate the wide variety of cargo coming into the 
United States and the inability to bring special nuclear material and other 
threat objects to ports of entry and place them in cargo during field 
validation. Similarly, while they acknowledged that injection studies have 
limitations, DOE national laboratory officials said the studies can increase 
the statistical confidence in comparisons of ASPs' and PVTs' probability of 
detecting threats concealed in cargo because of the possibility of 
supporting larger sample sizes than feasible with actual testing. A 
February 2008 DHS independent review team report on ASP testing also 
highlighted the benefits of injection studies, including the ability to 
explore ASP performance against a large number of threat scenarios at a 
practical cost and schedule and to permit an estimate of the minimum 
detectable amount for various threats. 16 



DHS Homeland Security Institute, Independent Review of the Department of Homeland 
Security Domestic Nuclear Detection Office Advanced Spectroscopic Portal: Final Report 
(Feb. 20, 2008). 
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DNDO has the data needed to conduct the studies. It has supported efforts 
to collect data on the radiation signatures for a variety of threat objects, 
including special nuclear materials, as recorded by both ASP and PVT 
systems. It has also collected about 7,000 usable "stream-of-commerce" 
records from ASP and PVT systems installed at a seaport. Furthermore, 
DNDO had earlier indicated that injection studies could provide 
information comparing the performance of the two systems as part of the 
certification process for both primary and secondary screening. However, 
addressing deficiencies in the stream-of-commerce data delayed the 
studies, and DNDO subsequently decided that performance testing would 
provide sufficient information to support a decision on ASP certification. 
DNDO officials said they would instead use injection studies to support 
effective deployment of the new portal monitors. 



Given that radiation detection equipment is already being used at ports of 
entry to screen for smuggled nuclear or radiological materials, the 
decision whether to replace existing equipment requires that the benefits 
of the new portal monitors be weighed against the costs. DNDO 
acknowledges that ASPs are significantly more expensive than PVTs to 
deploy and maintain, and based on preliminary results from the 2008 
testing, it is not yet clear that the $2 billion cost of DNDO's deployment 
plan is justified. Even if ASPs are able to reduce the volume of innocent 
cargo referred for secondary screening, they are not expected to detect 
certain nuclear materials that are surrounded by a realistic level of 
shielding better than PVTs could. Preliminary results of DNDO's 
performance testing show that ASPs outperformed the PVTs in detection 
of such materials during runs with light shielding, but ASPs' performance 
rapidly deteriorated once shielding was slightly increased. Furthermore, 
DNDO and DOE officials acknowledged that the performance of both 
portal monitors in detecting such materials with a moderate amount of 
shielding would be similarly poor. This was one of the reasons that 
performance testing did not include runs with a moderate level of 
shielding. 

Two additional aspects of the 2008 round of testing call into question 
whether ASPs' ability to provide a marginal improvement in detection of 
nuclear materials and reduce innocent alarms warrants the cost of the new 
technology. First, the DHS criteria for a significant increase in operational 
effectiveness do not take into account recent efforts to improve the 
current-generation portal monitors' sensitivity to nuclear materials 
through the "energy windowing" technique, most likely at a much lower 
cost. Data on developing this technique were collected during the 2008 
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round of performance testing but have not been analyzed. Second, while 
DNDO made improvements to the 2008 round of ASP testing that provided 
credibility to the test results, its test schedule does not allow for 
completion of injection studies prior to certification even though the 
studies could provide additional insight into the performance of the new 
technology. Without results from injection studies, the Secretary of 
Homeland Security would have to make a decision on certification based 
on a limited number of test scenarios conducted at the Nevada Test Site. 

Assuming that the Secretary of Homeland Security certifies ASPs, CBP 
officials anticipate that they will discover problems with the equipment 
when they start using it in the field. Integration testing uncovered a 
number of such problems, which delayed testing and resulted in ASP 
vendors making multiple changes to their systems. Correcting such 
problems in the field could prove to be more costly and time consuming 
than correcting problems uncovered through testing, particularly if DNDO 
proceeds directly from certification to full-scale deployment, as allowed 
under the congressional certification requirement that ASPs provide a 
significant increase in operational effectiveness. 



Recommendations for ^ e recommen d that the Secretary of Homeland Security direct the 

Director of DNDO to take the following two actions to ensure a sound 
Executive Action basis for a decision on ASP certification: 

• Assess whether ASPs meet the criteria for a significant increase in 
operational effectiveness based on a valid comparison with PVTs' full 
performance potential, including the potential to further develop PVTs' 
use of energy windowing to provide greater sensitivity to threats. Such a 
comparison could also be factored into an updated cost-benefit analysis to 
determine whether it would be more cost-effective to continue to use 
PVTs or deploy ASPs for primary screening at particular ports of entry. 

• Revise the schedule for ASP testing and certification to allow sufficient 
time for review and analysis of results from the final phases of testing and 
completion of all tests, including injection studies. 

If ASPs are certified, we further recommend that the Secretary of 
Homeland Security direct the Director of DNDO to develop an initial 
deployment plan that allows CBP to uncover and resolve any additional 
problems not identified through testing before proceeding to full-scale 
deployment — for example, by initially deploying ASPs at a limited number 
of ports of entry. 
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Agency Comments 
and Our Evaluation 



We provided a draft of this report to DOE and DHS for their review and 
comment. DOE provided technical comments, which we have 
incorporated into our report as appropriate. DHS's written comments are 
reproduced in appendix II. 

DHS agreed in part with our recommendations. Specifically, DHS stated 
that it believes its plan to deploy ASPs in phases, starting at a small 
number of low-impact locations, is in accordance with our 
recommendation to develop an initial deployment plan that allows 
problems to be uncovered and resolved prior to full-scale deployment. We 
agree that this deployment plan would address our recommendation and 
note that DHS's comments are the first indication provided to us of the 
department's intention to pursue such a plan. 

In contrast, DHS did not concur with our recommendations to (1) assess 
whether ASPs meet the criteria for a significant increase in operational 
effectiveness based on a comparison with PVTs' full potential, including 
further developing PVTs' use of energy windowing; and (2) revise the ASP 
testing and certification schedule to allow sufficient time for completion of 
all tests, including injection studies. With regard to energy windowing, 
DHS stated that using current PVT performance as a baseline for 
comparison is a valid approach because the majority of increased PVT 
performance through energy windowing has already been achieved. While 
DHS may be correct, its assessment is based on expert judgment rather 
than the results of testing and analysis being considered by the department 
to optimize the use of energy windowing. Given the marginal increase in 
sensitivity required of ASPs, we stand by our recommendation to assess 
ASPs against PVTs' full potential. DHS can then factor PVTs' full potential 
into a cost-benefit analysis prior to acquiring ASPs. On this point, DHS 
commented that its current cost-benefit analysis is a reasonable basis to 
guide programmatic decisions. However, upon receiving DHS's comments, 
we contacted DNDO to obtain a copy of its cost-benefit analysis and were 
told the analysis is not yet complete. 

With regard to injection studies, DHS agreed that the schedule for ASP 
certification must allow sufficient time for review and analysis of test 
results but stated that DHS and DOE experts concluded injection studies 
were not required for certification. DHS instead stated that the series of 
ASP test campaigns would provide a technically defensible basis for 
assessing the new technology against the certification criteria. However, 
DHS did not rebut the reasons we cited for conducting injection studies 
prior to certification, including test delays that have allowed more time to 
conduct the studies and the ability to explore ASP performance against a 
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large number of threat scenarios at a practical cost and schedule. On the 
contrary, DHS acknowledged the delays to testing and the usefulness of 
injection studies. Given that each phase of testing has revealed new 
information about the capabilities and limitations of ASPs, we believe 
conducting injection studies prior to certification would likely offer similar 
insights and would therefore be prudent prior to a certification decision. 

DHS provided additional comments regarding our assessment of the 
relative sensitivity of ASPs and PVTs and our characterization of the 
severity of the ASPs' software problems uncovered during field validation. 
With regard to sensitivity, DHS implied that our characterization of the 
relative ability of ASPs and PVTs is inaccurate and misleading because we 
did not provide a complete analysis of test results. We disagree. First, in 
meetings to discuss the preliminary results of performance testing 
conducted at the Nevada Test Site, DNDO officials agreed with our 
understanding of the ability of ASPs and PVTs deployed for primary 
screening to detect shielded nuclear materials. Furthermore, contrary to 
the assertion that a complete analysis requires a comparison of ASPs to 
handheld identification devices, our presentation is consistent with DHS's 
primary screening criterion for detection of shielded nuclear materials, 
which only requires that ASPs be compared with PVTs. Finally, while we 
agree that the performance test results require a more complete analysis, 
DNDO did not provide us with its final performance test report until early 
April 2009, after DHS provided its comments on our draft report. In the 
absence of the final report, which DNDO officials told us took longer than 
anticipated to complete, we summarized the preliminary results that 
DNDO presented to us during the course of our review as well as to 
congressional stakeholders. 

With regard to ASP software problems uncovered during field validation, 
we clarified our report in response to DHS's comment that the severity of 
the problems has not yet been determined. DHS stated that its preliminary 
analysis indicates the problems should be resolved by routine adjustments 
to threshold settings rather than presumably more significant software 
"revisions." However, given the history of lengthy delays during ASP 
testing, we believe that DHS's assessment of the severity of problems 
encountered during field validation may be overly optimistic. 



As agreed with your offices, unless you publicly announce the contents of 
this report earlier, we plan no further distribution until 30 days from the 
report date. At that time, we will send copies to the Secretaries of 
Homeland Security and Energy; the Administrator of NNSA; and interested 
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congressional committees. The report will also be available at no charge 
on the GAO Web site at http://www.gao.gov. 

If you or your staffs have any questions about this report, please contact 
me at (202) 512-3841 or aloisee@gao.gov. Contact points for our Offices of 
Congressional Relations and Public Affairs may be found on the last page 
of this report. GAO staff who made key contributions to this report are 
listed in appendix III. 




Gene Aloise 

Director, Natural Resources 
and Environment 
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Appendix I: Scope and Methodology 



To evaluate the degree to which Department of Homeland Security's 
(DHS) criteria for a significant increase in operational effectiveness 
address the limitations of the current generation of radiation detection 
equipment, we clarified the intent of the criteria through the Domestic 
Nuclear Detection Office's (DNDO) written answers to our questions and 
through interviews with U.S. Customs and Border Protection (CBP) 
officials. We also took steps to gain a fuller understanding of the strengths 
and limitations of the current-generation equipment, which the criteria use 
as a baseline for evaluating the effectiveness of advanced spectroscopic 
portals (ASP). In particular, we obtained copies of the Department of 
Energy (DOE) threat guidance and related documents used to set 
polyvinyl toluene (PVT) thresholds for detection of nuclear materials. We 
interviewed DOE and national laboratory officials responsible for the 
threat guidance about the process for developing it and the basis for its 
underlying assumptions, including shielding levels. We also interviewed 
DNDO and Pacific Northwest National Laboratory officials regarding the 
extent to which PVTs currently deployed at ports of entry meet the 
guidance and the development and use of energy windowing to enhance 
PVTs' sensitivity to nuclear materials. 

To evaluate the rigor of the 2008 round of testing as a basis for 
determining ASPs' operational effectiveness, we reviewed the test and 
evaluation master plan and plans for individual phases of testing, including 
system qualification testing conducted at vendors' facilities, performance 
testing conducted at the Nevada Test Site for evaluating ASP detection and 
identification capabilities, and integration testing conducted at Pacific 
Northwest National Laboratory for evaluating the readiness of ASPs to be 
used in an operational environment at ports of entry. We also reviewed 
draft plans for field validation conducted at CBP ports of entry and the 
DHS Science and Technology Directorate's independent operational test 
and evaluation. In reviewing these documents, we specifically evaluated 
the extent to which the performance test design was sufficient to identify 
statistically significant differences between the ASP and current- 
generation systems and whether DHS had addressed our concerns about 
previous rounds of ASP testing. We interviewed DNDO, CBP, and other 
DHS officials responsible for conducting and monitoring tests, and we 
observed, for one day each, performance testing at the Nevada Test Site 
and integration testing at DOE's Pacific Northwest National Laboratory. 
We also interviewed representatives of entities that supported testing, 
including DOE's National Nuclear Security Administration and Pacific 
Northwest National Laboratory, the National Institute of Standards and 
Technology, and the Johns Hopkins University Applied Physics 
Laboratory. We reviewed the DHS independent review team report of 
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Appendix I: Scope and Methodology 



previous ASP testing conducted in 2007, and we interviewed the chair of 
the review team to clarify the report's findings. Finally, we examined 
preliminary or final results for the phases of testing completed during our 
review, and we interviewed DNDO and CBP officials regarding the results. 

To evaluate the test schedule, we analyzed the initial working schedule 
DNDO provided to us in May 2008 and the schedule presented in the 
August 2008 test and evaluation master plan, and we tracked changes to 
the schedule and the reasons for any delays. We interviewed DNDO and 
other officials with a role in testing to determine the amount of time 
allowed for analysis and review of results. We interviewed DNDO and 
Pacific Northwest National Laboratory officials regarding the injection 
studies, including reasons for delays in the studies and plans for including 
the results as part of the ASP certification process. 

We conducted this performance audit from May 2008 to May 2009 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. 
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Appendix II: Comments from the Department 
of Homeland Security 



U.S. Department of Homeland Security 

Washington, DC 20528 

#Sfe. Homeland 
ifP Security 

March 24, 2009 

Mr. Gene Aloise 

Director, Natural Resources and Environment 
U.S Government Accountability Office 
441 G Street NW 
Washington, DC 20548 

Dear Mr. Aloise: 

RE: Draft Report GAO-09-354SU, Combating Nuclear Terrorism: DHS Improved 
Testing of Advanced Radiation Detection Portal Monitors, But Preliminary Results 
Show Limits of the New Technology (360961) 

The Department of Homeland Security (DHS) appreciates the opportunity to comment on the 
above referenced draft report and clarify several points. DHS is committed to rigorous testing 
prior to the deployment of new technologies and is pleased that the Government 
Accountability Office (GAO) recognizes improvements that the Domestic Nuclear Detection 
Office (DNDO) has incorporated into its testing programs. 

The comments provided below in response to the subject report are restricted to points that are 
directly relevant to the three areas that GAO was asked to review: (1) the degree to which 
DHS's criteria for significant increase in operational effectiveness address the limitations of 
existing radiation detection equipment, (2) the rigor of Advanced Spectroscopic Portal (ASP) 
testing and preliminary test results, and (3) the ASP test schedule. 

Report Recommendation: "Assess whether ASPs meet the criteria for significant increase in 
operational effectiveness based on a valid comparison with PVTs 'full performance potential, 
including the potential to further develop PVTs ' use of energy windowing to provide greater 
sensitivity to threats. Such a comparison could also be factored in to an updated cost-benefit 
analysis to determine whether it would be more cost-effective to continue to use PVTs or 
deploy ASPs for primary screening at particular ports of entry, " 

DHS does not concur with this recommendation. 

It is the judgment of experts that the majority of any increased Polyvinyl Toluene (PVT) 
performance from energy windowing has already been achieved in the existing systems; 
therefore we are confident that using the current understanding on PVT energy windowing as 
a baseline is a valid approach. DHS believes that Customs and Border Protection (CBP), the 
Office of the Undersecretary for Management, and DNDO have collaborated to establish a 
meaningful set of criteria to measure the relative operational effectiveness of the current ASP 
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and PVT systems. Therefore, the current cost-benefit analysis is a reasonable basis to guide 
programmatic decisions. However, given that PVT systems will continue to be operated in the 
field, DNDO has initiated efforts to continuously improve the capability of the PVT systems. 
These efforts include an ongoing analysis of potential optimization of the PVT energy 
windowing capability that will proceed in parallel with other activities related to Certification. 

Report Recommendation: "Revise the schedule for ASP testing and certification to allow 
sufficient time for review and analysis of results from the final phases of testing and 
completion of all tests, including injection studies ". 

DHS does not concur with aspects of this recommendation. 

We agree that there must be sufficient time to review the analyses and results. The current 
schedule for certification is predicated on completing all of the requisite testing and analysis 
of results required to satisfy the criteria for demonstrating significantly improved operational 
effectiveness. DHS has demonstrated that it is implementing a deliberate process to ensure all 
the pre-established exit criteria have been achieved prior to concluding the relevant phase of 
testing. 

However, a team of subject matter experts from the Department of Energy (DOE) and DHS 
concluded that injection studies were not required for Certification. The basis of this decision 
was a review of the series of test campaigns planned for ASP and the joint determination that 
these tests would provide a technically defensible basis for assessing the ASP technology 
against the Certification criteria. "Accordingly, Certification will be based on the whole body 
of experimental knowledge collected on ASP, including technical performance at Nevada 
Test Site (NTS), integration testing at 331-G, field validation at ports of entry, reanalysis of 
selected raw data files through a validated replay tool, and independent Operational Test & 
Evaluation. Nevertheless, DHS believes that injection studies are useful and will continue to 
support and perform them as an aid for deployment decisions and for future development. 

Regarding the conclusions section in your report, we offer the following additional comments: 

• Conclusions regarding the relative effectiveness of PVT and ASP to detect Highly 
Enriched Uranium (HEU) require a more complete analysis than given in your report. 
For example, the effectiveness of a system should include the entire system (primary 
plus secondary inspection). Such an analysis would necessarily incorporate a 
comparison of ASP to a hand-held RIID. Without including the whole system, 
conclusions about overall detection effectiveness are inaccurate and misleading. 

• We agree with CBP that we will inevitably discover problems with this highly 
advanced and complex system as it transitions into field operation. There is simply no 
way to fully anticipate and replicate all real- world problems before deployment to the 
field: computer modeling and simulation does not provide the same level of 
experience or exposure. In anticipation of this fact, deployment will be accomplished 
in phases, starting with a small number of low impact locations. As the issues with the 
system are corrected, we will gradually build up to wider deployment. The process 
will be similar to what was used in the successful deployment of PVTs. Therefore, 
DHS believes ASP deployment will be implemented in accordance with your 



2 



Page 26 



GAO-09-655 Combating Nuclear Smuggling 



Appendix II: Comments from the Department 
of Homeland Security 



recommendation to the Secretary to discover and resolve problems through initial 
deployment to a limited number of locations. 

Finally, there were a few conclusions in the body of your report that we would like to 
comment on: 

• We agree that the NTS test did not "represent the full range of possibilities for 
concealing smuggled nuclear or radiological materials" because such a range is 
unpractically large to create. However, the NTS test campaign included an extensive 
array of shielding and masking configurations in a plan designed jointly by DHS and 
DOE to cover a range relevant to a passive radiation scanning application. 

• Your report states that "CBP suspended field validation after one week because of 
serious performance problems requiring software revisions." Although it remains for 
analysis to determine the severity of the problems encountered, preliminary analysis 
indicates that the problems should be resolved by making adjustments to threshold 
settings. Such adjustments are part of any installation of an RPM (PVT or ASP) and 
are not software revisions. As mentioned above, it is impossible to anticipate all the 
problems that will occur in real-world operation, so it is not surprising to encounter 
problems, given that this is the first opportunity for the latest version of ASP to 
operate in the flow of real commerce. 



Thank you for the opportunity to review and provide comments on your report. We look 
forward to working with you on future homeland security issues. 



Sincerely, 




Director 

Departmental GAO/OIG Liaison Office 
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